{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 基于 Bi-directional LSTM 的序列标注任务（分词）\n",
    "\n",
    "\n",
    "**tensorflow 版本： 1.2.1**\n",
    "\n",
    "\n",
    "本例子主要参考[【中文分词系列】 4. 基于双向LSTM的seq2seq字标注]{url: http://spaces.ac.cn/archives/3924/} 这篇文章。<br/>\n",
    "该文章用的是 keras 实现的双端 LSTM，在本例中，实现思路和该文章基本上一样，只是用 TensorFlow 来实现的。<br/>\n",
    "\n",
    "本例最主要的是说明基于 TensorFlow 如何来实现 Bi-LSTM。在后面部分进行最后分词处理用的是维特比译码，如果想了解为什么的话可以看一下《统计学习方法》第10章介绍的隐马尔可夫模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "主要参考: <br/>\n",
    "[1] 【中文分词系列】 4. 基于双向LSTM的seq2seq字标注 http://spaces.ac.cn/archives/3924/  <br/>\n",
    "[2] https://github.com/yongyehuang/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/bidirectional_rnn.py  <br/>\n",
    "[3] https://github.com/yongyehuang/deepnlp/blob/master/deepnlp/pos/pos_model_bilstm.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "from tqdm import tqdm\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of texts is 20339269\n",
      "Example of texts: \n",
      "“/s  人/b  们/e  常/s  说/s  生/b  活/e  是/s  一/s  部/s  教/b  科/m  书/e  ，/s  而/s  血/s  与/s  火/s  的/s  战/b  争/e  更/s  是/s  不/b  可/m  多/m  得/e  的/s  教/b  科/m  书/e  ，/s  她/s  确/b  实/e  是/s  名/b  副/m  其/m  实/e  的/s  ‘/s  我/s  的/s  大/b  学/e  ’/s  。/s  \n",
      "“/s  心/s  静/s  渐/s  知/s  春/s  似/s  海/s  ，/s  花/s  深/s  每/s \n"
     ]
    }
   ],
   "source": [
    "# 以字符串的形式读入所有数据\n",
    "with open('raw_data/msr_train.txt', 'rb') as inp:\n",
    "    texts = inp.read().decode('gbk')\n",
    "sentences = texts.split('\\r\\n')  # 根据换行切分\n",
    "\n",
    "# 将不规范的内容（如每行的开头）去掉\n",
    "def clean(s): \n",
    "    if u'“/s' not in s:  # 句子中间的引号不应去掉\n",
    "        return s.replace(u' ”/s', '')\n",
    "    elif u'”/s' not in s:\n",
    "        return s.replace(u'“/s ', '')\n",
    "    elif u'‘/s' not in s:\n",
    "        return s.replace(u' ’/s', '')\n",
    "    elif u'’/s' not in s:\n",
    "        return s.replace(u'‘/s ', '')\n",
    "    else:\n",
    "        return s\n",
    "    \n",
    "texts = u''.join(map(clean, sentences)) # 把所有的词拼接起来\n",
    "print 'Length of texts is %d' % len(texts)\n",
    "print 'Example of texts: \\n', texts[:300]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentences number: 332856\n",
      "Sentence Example:\n",
      "  人/b  们/e  常/s  说/s  生/b  活/e  是/s  一/s  部/s  教/b  科/m  书/e  \n"
     ]
    }
   ],
   "source": [
    "# 重新以标点来划分\n",
    "sentences = re.split(u'[，。！？、‘’“”]/[bems]', texts)\n",
    "print 'Sentences number:', len(sentences)\n",
    "print 'Sentence Example:\\n', sentences[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "12667it [00:00, 60893.12it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start creating words and tags data ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "332856it [00:04, 71297.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of datas is 321827\n",
      "Example of datas:  [u'\\u4eba' u'\\u4eec' u'\\u5e38' u'\\u8bf4' u'\\u751f' u'\\u6d3b' u'\\u662f'\n",
      " u'\\u4e00' u'\\u90e8' u'\\u6559' u'\\u79d1' u'\\u4e66']\n",
      "Example of labels: [u'b' u'e' u's' u's' u'b' u'e' u's' u's' u's' u'b' u'm' u'e']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "def get_Xy(sentence):\n",
    "    \"\"\"将 sentence 处理成 [word1, w2, ..wn], [tag1, t2, ...tn]\"\"\"\n",
    "    words_tags = re.findall('(.)/(.)', sentence)\n",
    "    if words_tags:\n",
    "        words_tags = np.asarray(words_tags)\n",
    "        words = words_tags[:, 0]\n",
    "        tags = words_tags[:, 1]\n",
    "        return words, tags # 所有的字和tag分别存为 data / label\n",
    "    return None\n",
    "\n",
    "datas = list()\n",
    "labels = list()\n",
    "print 'Start creating words and tags data ...'\n",
    "for sentence in tqdm(iter(sentences)):\n",
    "    result = get_Xy(sentence)\n",
    "    if result:\n",
    "        datas.append(result[0])\n",
    "        labels.append(result[1])\n",
    "\n",
    "print 'Length of datas is %d' % len(datas) \n",
    "print 'Example of datas: ', datas[0]\n",
    "print 'Example of labels:', labels[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tags</th>\n",
       "      <th>words</th>\n",
       "      <th>sentence_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[b, e, s, s, b, e, s, s, s, b, m, e]</td>\n",
       "      <td>[人, 们, 常, 说, 生, 活, 是, 一, 部, 教, 科, 书]</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[s, s, s, s, s, b, e, s, s, b, m, m, e, s, b, ...</td>\n",
       "      <td>[而, 血, 与, 火, 的, 战, 争, 更, 是, 不, 可, 多, 得, 的, 教, ...</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                tags  \\\n",
       "0               [b, e, s, s, b, e, s, s, s, b, m, e]   \n",
       "1  [s, s, s, s, s, b, e, s, s, b, m, m, e, s, b, ...   \n",
       "\n",
       "                                               words  sentence_len  \n",
       "0               [人, 们, 常, 说, 生, 活, 是, 一, 部, 教, 科, 书]            12  \n",
       "1  [而, 血, 与, 火, 的, 战, 争, 更, 是, 不, 可, 多, 得, 的, 教, ...            17  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data = pd.DataFrame({'words': datas, 'tags': labels}, index=range(len(datas)))\n",
    "#　句子长度\n",
    "df_data['sentence_len'] = df_data['words'].apply(lambda words: len(words))\n",
    "df_data.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAEXCAYAAADvDECpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2cVdV97/HPN6BIfESlVAHBRjItYp5MlaS3udOSCokm\n0Fsf6DUJJCS8cjVqUtIUalLNA602TdLYVhsSjWgSlRBzpbFGKWZi0xQIJhpFQ+QKBhBFRRE0ouDv\n/rHWcTYnZ2YOM+fMHjjf9+t1XrP32nutvfY6Z85v9tpr9lJEYGZmVpZXlV0BMzNrbQ5EZmZWKgci\nMzMrlQORmZmVyoHIzMxK5UBkZmalciAyJP2rpE81qKzjJO2QNCivd0j6YCPKzuXdJmlGo8rbi+N+\nTtKTkh6rc/9LJX2j2fXqT5KulfS5BpXVJukeSdslXdiIMm3f5UC0n5O0XtKv8y/8M5J+LOnDkl55\n7yPiwxHx2TrLent3+0TEryLikIjY3YC6/8aXeUS8IyIW9rXsvazHccAcYHxE/HaN7e2SNjbx+A0L\nAHtxzJmSftTEQ3wC+EFEHBoRV9Q4/omS7pC0NX9u75b0zr4etNnvlfWOA1FreFdEHAqMAS4D/gq4\nutEHkTS40WUOEMcBT0XElrIrsh8ZA6zuZvu/AUuB3wZ+C7gQeLYf6mVliAi/9uMXsB54e1XaKcDL\nwIS8fi3wubx8NPA94BlgK/CfpD9Yrs95fg3sIP1FOxYIYBbwK+CuQtrgXF4H8HfAStIXyS3AkXlb\nO7CxVn2BKcCLwEv5ePcWyvtgXn4V8EngEWALcB1weN5WqceMXLcngYu7aafDc/4ncnmfzOW/PZ/z\ny7ke11blO7hq+w7gWOBSYFEuczvpS/fNhXzHAt/Jx1sHXNhN3V55f2ps+13SF/ZWYA1wdlW+fwFu\nzXVYAbymsP20nGcbcCXwQ+CDwO8BLwC78/k8U095Ner27nzez+T37fdy+p257Bdy+a+tynd0fu+O\n6KbsM4B7ctk/Bl5X9Rn6OPDzfG43AQd18169CpgL/D/gqfy+VT6j3X6OgEHAX+e824G7gdE9vTd+\nVb2fZVfArya/wTUCUU7/FfB/8vIrX3SkoPGvwAH59YeAapVV+CW9Lv+SD6V2INoETMj7fAf4Rt7W\nTheBKC9fWtm3sL2DzkD0AWAt8DvAIcDNwPVVdftqrtfrgZ3kL8Ma7XEdKUgemvP+EpjVVT2r8tY6\nj0tJX7TvzF9Wfwcsz9telb+w/gY4MNf/YWByF+W/8v5UpR8MbADeDwwG3kj6ohxfyPcU6Q+PwcA3\ngRvztqNJfxj8r7ztIlLQr7TtTOBHNepRs7wadXst8BzwJ6TP0Sfye3Vg9ftYI6+Ah0h/EE0DRlRt\nfyPpD49Tc9vOyJ+bIYXP0EpSkDkSeBD4cDfv1UXAcmAUMAT4CnBDPZ8j4C+B+4C2XO/XA0f19N74\ntefLXXOt61HSL2m1l4BjgDER8VJE/Gfk37huXBoRz0XEr7vYfn1E3B8RzwGfAs6uDGboo3OBL0bE\nwxGxA5gHTK/qIvx0RPw6Iu4F7iV9Uewh12U6MC8itkfEeuALwHv7WL8fRcS/R7pfdn3h2L8PDI+I\nz0TEixHxMOmLbvpeln8GsD4ivh4RuyLiZ6RAf1Zhn+9GxMqI2EUKHG/I6e8EVkfEzXnbFUA9AzG6\nKq/aOcCtEbE0Il4C/oH0Rf7Wng6QP29/RAooXwA2S7pL0ri8y2zgKxGxIiJ2R7pnuBOYWCjmioh4\nNCK2krr5uqonwIdJVzkbI2In6Y+IM+v8HH0Q+GRErInk3oh4ivreG8v21z5969lIUpdBtc+TfhHv\nkASwICIu66GsDXux/RHSX8hH11fNbh2byyuWPRgYUUgrfrk+T7pyqnZ0rlN1WSP7WL/qYx+Uv9zG\nAMdKeqawfRCpG3RvjAFOrSpnMCnodVWHyvkfS+F9iYio8yZ+Pe1ZKf+V9oyIlyVtoM42jYiNwEcA\nJI0GFpCuWt9COu8Zki4oZDkwH7Oreha3VRsDfFfSy4W03dT3ORpN6parVWZP741lDkQtSNLvk74Q\nfmNUVERsJ40QmyNpAnCnpJ9ExDJSF0UtPV0xjS4sH0e66nqS1HXz6kK9BgHD96LcR0m/8MWydwGP\nk7pZ6vVkrtMY4IFCWZvqzL+3j7DfAKyLiHE97tlzOT+MiD/pRd7NFNpI6a+OYpv19bH8jwInVZU/\nmvrbtLMiERsk/QtwQ07aAMyPiPm9qFet89oAfCAi/qt6g6SxPZS3AXgNcH+N9N6+Ny3HXXMtRNJh\nks4AbiTde7mvxj5nSDohf3FsI/1lWPlL8XHS/Yy99R5J4yW9GvgMsDh3V/2SdJVwuqQDSAMEhhTy\nPQ6MLQ41r3ID8DFJx0s6BPhb4KbcbVS3XJdFwHxJh0oaA/wFUO//AT0OHCXp8Dr3Xwlsl/RXkoZK\nGiRpQv4DoSuDJB1UeB1IuofyWknvlXRAfv2+pN+row63AidJmpav0s4njVArntOofJzeWAScLmlS\nfm/nkLrPftxTRknDJH06fw5fJelo0v3A5XmXrwIflnSqkoPzZ+jQOupV6736V9J7PyYff7ikqXWe\n59eAz0oal+vyOklH0bf3puU4ELWGf5O0nfRX2sXAF0k3UWsZB/wHaUTRfwNXRsQP8ra/Az6Z/6/j\n43tx/OtJN7ofI41euhAgIrYB55F+mTeRrpCK3UPfzj+fkvTTGuVek8u+izTy7AXgghr71eOCfPyH\nSVeK38rl9ygifkEKig/ntumuG6gS+M4g3bdYR7oi+xpp5F5X5pJGfFVed+ar19NI95YeJbXv5ewZ\nzLuqw5Ok+xV/TxqAMB5YRQoWkEa2rQYek/RkT+XVKH8N8B7gn0jn9y7SvxG8WEf2F0mDBP6DNKDi\n/lyvmbnsVcCHgH8GniYNgphZZ71qvVdfBpaQuqO3kwLeqfWUR/pdWgTcket6NTC0L+9NK6qMhjKz\nFpavOjcC5xb+8DDrF74iMmtRkiZLOkLSENL/wojO7i+zfuNAZNa63kIa8VXpOpvWzRB8s6Zx15yZ\nmZXKV0RmZlYq/x9RwRFHHBEnnHBC2dUYEJ577jkOPvjgsqsxILgtOrktOrktOt19991PRsTwnves\nzYGoYMSIEaxatarsagwIHR0dtLe3l12NAcFt0clt0clt0UnSIz3v1TV3zZmZWakciMzMrFQORGZm\nVioHIjMzK5UDkZmZlcqByMzMSuVAZGZmpWpqIJJ0jaQtku4vpH1e0i8k/VzSdyUdUdg2T9JaSWsk\nTS6knyzpvrztijxXDpKGSLopp68oTmIlaYakh/JrRjPP08zMeq/ZV0TXAlOq0pYCEyLidaSJ0eYB\nSBpPmrvjxJznyjxjJ8BVpPlHxuVXpcxZwNMRcQLwJdJ8H0g6EriENKfIKcAlkoY14fzMzKyPmvpk\nhYi4q3qq3Yi4o7C6HDgzL08FboyIncA6SWuBUyStBw6LiOUAkq4DpgG35TyX5vyLgX/OV0uTgaUR\nsTXnWUoKXpWphns0du6tdZ9nvdZfdnrDyzQz29eVfY/oA6SAAjCSNINoxcacNpI9Z+2spO+RJ08P\nvQ04qpuyzMxsgCntWXOSLgZ2Ad8sqw65HrOB2QDDhw+no6MDgDkn7Wr4sSpl7wt27NixT9W3mdwW\nndwWndwWjVNKIJI0EzgDmBSdEyJtAkYXdhuV0zbl5er0Yp6NkgYDhwNP5fT2qjwdteoSEQuABQBt\nbW1ReYjhzGZ0zZ3b3uM+A4Uf6NjJbdHJbdHJbdE4/d41J2kK8Ang3RHxfGHTEmB6Hgl3PGlQwsqI\n2Aw8K2livv/zPuCWQp7KiLgzgTtzYLsdOE3SsDxI4bScZmZmA0xTr4gk3UC6Mjla0kbSSLZ5wBBg\naR6FvTwiPhwRqyUtAh4gddmdHxG7c1HnkUbgDSXdU6rcV7oauD4PbNhKGnVHRGyV9FngJ3m/z1QG\nLpiZ2cDS7FFzf14j+epu9p8PzK+RvgqYUCP9BeCsLsq6Brim7sqamVkpyh41Z2ZmLc6ByMzMSuVA\nZGZmpXIgMjOzUjkQmZlZqRyIzMysVA5EZmZWKgciMzMrlQORmZmVyoHIzMxK5UBkZmalciAyM7NS\nORCZmVmpHIjMzKxUDkRmZlYqByIzMyuVA5GZmZXKgcjMzErlQGRmZqVyIDIzs1I5EJmZWakciMzM\nrFQORGZmVioHIjMzK5UDkZmZlcqByMzMStXUQCTpGklbJN1fSDtS0lJJD+Wfwwrb5klaK2mNpMmF\n9JMl3Ze3XSFJOX2IpJty+gpJYwt5ZuRjPCRpRjPP08zMeq/ZV0TXAlOq0uYCyyJiHLAsryNpPDAd\nODHnuVLSoJznKuBDwLj8qpQ5C3g6Ik4AvgRcnss6ErgEOBU4BbikGPDMzGzgaGogioi7gK1VyVOB\nhXl5ITCtkH5jROyMiHXAWuAUSccAh0XE8ogI4LqqPJWyFgOT8tXSZGBpRGyNiKeBpfxmQDQzswFg\ncAnHHBERm/PyY8CIvDwSWF7Yb2NOeykvV6dX8mwAiIhdkrYBRxXTa+TZg6TZwGyA4cOH09HRAcCc\nk3bt/Zn1oFL2vmDHjh37VH2byW3RyW3RyW3ROGUEoldEREiKkuuwAFgA0NbWFu3t7QDMnHtrw4+1\n/tz2hpfZLB0dHVTaotW5LTq5LTq5LRqnjFFzj+fuNvLPLTl9EzC6sN+onLYpL1en75FH0mDgcOCp\nbsoyM7MBpoxAtASojGKbAdxSSJ+eR8IdTxqUsDJ34z0raWK+//O+qjyVss4E7sz3kW4HTpM0LA9S\nOC2nmZnZANPUrjlJNwDtwNGSNpJGsl0GLJI0C3gEOBsgIlZLWgQ8AOwCzo+I3bmo80gj8IYCt+UX\nwNXA9ZLWkgZFTM9lbZX0WeAneb/PRET1oIl+N7bB3X3rLzu9oeWZmZWhqYEoIv68i02Tuth/PjC/\nRvoqYEKN9BeAs7oo6xrgmrora2ZmpfCTFczMrFQORGZmVioHIjMzK5UDkZmZlcqByMzMSuVAZGZm\npXIgMjOzUjkQmZlZqRyIzMysVA5EZmZWKgciMzMrlQORmZmVyoHIzMxK5UBkZmalciAyM7NSORCZ\nmVmpHIjMzKxUDkRmZlYqByIzMyuVA5GZmZXKgcjMzErlQGRmZqUaXM9Okl4HjC3uHxE3N6lOZmbW\nQnoMRJKuAV4HrAZezskBOBCZmVmf1XNFNDEixje9JmZm1pLquUf035IaHogkfUzSakn3S7pB0kGS\njpS0VNJD+eewwv7zJK2VtEbS5EL6yZLuy9uukKScPkTSTTl9haSxjT4HMzPru3oC0XWkYLRG0s/z\nl/7P+3JQSSOBC4E3R8QEYBAwHZgLLIuIccCyvE4OhNOBE4EpwJWSBuXirgI+BIzLryk5fRbwdESc\nAHwJuLwvdTYzs+aoJxBdDbyX9AX/LuCM/LOvBgNDJQ0GXg08CkwFFubtC4FpeXkqcGNE7IyIdcBa\n4BRJxwCHRcTyiAhS0CzmqZS1GJhUuVoyM7OBo557RE9ExJJGHjQiNkn6B+BXwK+BOyLiDkkjImJz\n3u0xYEReHgksLxSxMae9lJer0yt5NuTj7ZK0DTgKeLJYF0mzgdkAw4cPp6OjA4A5J+3q+4k2WaWu\nzbBjx46mlr8vcVt0clt0cls0Tj2B6GeSvgX8G7CzktiX4dv53s9U4HjgGeDbkt5T3CciQlL09hj1\niogFwAKAtra2aG9vB2Dm3Fubfeg+W39ue9PK7ujooNIWrc5t0clt0clt0Tj1BKKhpAB0WiGtr8O3\n3w6si4gnACTdDLwVeFzSMRGxOXe7bcn7bwJGF/KPymmb8nJ1ejHPxtz9dzjwVB/qbGZmTdBjIIqI\n9zfhuL8CJkp6NalrbhKwCngOmAFcln/ekvdfAnxL0heBY0mDElZGxG5Jz0qaCKwA3gf8UyHPDOC/\ngTOBO/N9JDMzG0Dq+YfWr5OugPYQER/o7UEjYoWkxcBPgV3Az0jdY4cAiyTNAh4Bzs77r5a0CHgg\n739+ROzOxZ0HXEu6crstvyANsrhe0lpgK2nUnZmZDTD1dM19r7B8EPCnpBFufRIRlwCXVCXvJF0d\n1dp/PjC/RvoqYEKN9BeAs/paTzMza656uua+U1yXdAPwo6bVyMzMWkpvnr49DvitRlfEzMxaUz33\niLaT7hEp/3wM+Ksm18vMzFpEPV1zh/ZHRczMrDXVOx/RSGAMe85HdFezKmVmZq2jnq65y4FzSEOn\nK0OmA3AgMjOzPqvnimga0BYRO3vc08zMbC/VM2ruYeCAZlfEzMxaUz1XRM8D90haxp4PPb2wabUy\nM7OWUU8gWpJfZmZmDVfP8O2F3W2X9J2I+LPGVcnMzFpJb56sUO13GlCGmZm1qEYEIk+tYGZmvdaI\nQGRmZtZrjQhEakAZZmbWouoKRJKGSmrrYrMfgGpmZr3WYyCS9C7gHuD7ef0Nkl4Zzh0RdzSvemZm\ntr+r54roUuAU4BmAiLgHOL6JdTIzsxZSTyB6KSK2VaV5pJyZmTVEPU9WWC3pfwODJI0DLgR+3Nxq\nmZlZq6jniugC4ETSc+a+BWwDPtrMSpmZWeuo5xE/zwMX55eZmVlD1TNqbqmkIwrrwyTd3txqmZlZ\nq6ina+7oiHimshIRTwO/1bwqmZlZK6knEL0s6bjKiqQxeNScmZk1SD2B6GLgR5Kul/QN4C5gXl8P\nLOkISYsl/ULSg5LeIunI3BX4UP45rLD/PElrJa2RNLmQfrKk+/K2KyQppw+RdFNOXyFpbF/rbGZm\njddjIIqI7wNvAm4CbgROjohG3CP6MvD9iPhd4PXAg8BcYFlEjAOW5XUkjQemk0bvTQGulDQol3MV\n8CFgXH5NyemzgKcj4gTgS8DlDaizmZk1WL0PPR0CbAWeBcZLeltfDirpcOBtwNUAEfFivg81FahM\nxLcQmJaXpwI3RsTOiFgHrAVOkXQMcFhELI+IAK6rylMpazEwqXK1ZGZmA0ePw7clXQ6cA6wGXs7J\nQeqi663jgSeAr0t6PXA3cBEwIiI2530eA0bk5ZHA8kL+jTntpbxcnV7JswEgInZJ2gYcBTxZdX6z\ngdkAw4cPp6OjA4A5J+3qw+n1j0pdm2HHjh1NLX9f4rbo5Lbo5LZonHqerDANaIuInQ0+7puACyJi\nhaQvk7vhKiIiJDV9UERELAAWALS1tUV7ezsAM+fe2uxD99n6c9ubVnZHRweVtmh1botObotObovG\nqadr7mHggAYfdyOwMSJW5PXFpMD0eO5uI//ckrdvAkYX8o/KaZvycnX6HnkkDQYOB55q8HmYmVkf\n1ROIngfukfSVPCrtCklX9OWgEfEYsKEwx9Ek4AFgCTAjp80AbsnLS4DpeSTc8aRBCStzN96zkibm\n+z/vq8pTKetM4M58H8nMzAaQerrmluRXo10AfFPSgaSrrveTAuMiSbOAR4CzASJitaRFpGC1Czg/\nInbncs4DrgWGArflF6SBENdLWksaaDG9CedQqrFN6D5cf9npDS/TzKw79TxrbqGkocBxEbGmUQfO\n8xq9ucamSV3sPx+YXyN9FTChRvoLwFl9rKaZmTVZn2doNTMz64veztD6O02sk5mZtZDeztD6cs09\nzczM9pJnaDUzs1L1dobWi5pZKTMzax31XBGdHhF7zNAq6Szg202rlZmZtYx6rohqTfnQ52kgzMzM\noJsrIknvAN4JjKx6ksJhpH8qNTMz67PuuuYeBVYB7yY9HbtiO/CxZlbKzMxaR5eBKCLuBe6V9K2I\neKkf62RmZi2knsEKp0i6FBiT9xdplgb/U6uZmfVZPYHoalJX3N3A7h72NTMz2yv1BKJtEXFbz7uZ\nmZntvXoC0Q8kfR64mfRPrQBExE+bViszM2sZ9QSiU/PP4pQNAfxx46tjZmatpp75iP6oPypiZmat\nqZ75iEZIulrSbXl9fJ5B1czMrM/qecTPtcDtwLF5/ZfAR5tVITMzay31BKKjI2IReQ6iiNiFh3Gb\nmVmD1BOInpN0FGmAApImkqaCMDMz67N6Rs39BbAEeI2k/wKGA2c2tVZmZtYy6rkieg3wDuCtpHtF\nD1FfADMzM+tRPYHoUxHxLDAM+CPgSuCqptbKzMxaRj2BqDIw4XTgqxFxK3Bg86pkZmatpJ5AtEnS\nV4BzgH+XNKTOfGZmZj2qJ6CcTbo3NDkingGOBP6yrweWNEjSzyR9L68fKWmppIfyz2GFfedJWitp\njaTJhfSTJd2Xt10hSTl9iKSbcvoKSWP7Wl8zM2uOHgNRRDwfETdHxEN5fXNE3NGAY18EPFhYnwss\ni4hxwLK8jqTxwHTgRGAKcKWkQTnPVcCHgHH5NSWnzwKejogTgC8BlzegvmZm1gSldLFJGkW65/S1\nQvJUYGFeXghMK6TfGBE7I2IdsJY0Wd8xwGERsTwiAriuKk+lrMXApMrVkpmZDSxlDcP+R+ATwKGF\ntBERsTkvPwaMyMsjgeWF/TbmtJfycnV6Jc8GSE+CkLQNOAp4sroikmYDswGGDx9OR0cHAHNO2tW7\nM9vHVc5/x44dryy3OrdFJ7dFJ7dF4/R7IJJ0BrAlIu6W1F5rn4gISdEf9YmIBcACgLa2tmhvT1Wa\nOffW/jj8gLP+3HYgBaRKW7Q6t0Unt0Unt0XjlHFF9AfAuyW9EzgIOEzSN4DHJR0TEZtzt9uWvP8m\nYHQh/6ictikvV6cX82yUNBg4HHiqWSdkZma91+/3iCJiXkSMioixpEEId0bEe0iPEZqRd5sB3JKX\nlwDT80i440mDElbmbrxnJU3M93/eV5WnUtaZ+Rj9coVlZmZ7ZyA9qucyYFGe6+gR0rBxImK1pEXA\nA8Au4PyIqPyT7XmkaSqGArflF8DVwPWS1gJbSQHPzMwGoFIDUUR0AB15+SlgUhf7zQfm10hfBUyo\nkf4CcFYDq2pmZk3iJySYmVmpHIjMzKxUDkRmZlYqByIzMyuVA5GZmZXKgcjMzErlQGRmZqVyIDIz\ns1I5EJmZWakciMzMrFQORGZmVioHIjMzK5UDkZmZlcqByMzMSuVAZGZmpXIgMjOzUjkQmZlZqRyI\nzMysVKVOFW4Dz9i5twIw56RdzMzLfbH+stP7XIaZ7d98RWRmZqVyIDIzs1I5EJmZWakciMzMrFQO\nRGZmVioHIjMzK1UpgUjSaEk/kPSApNWSLsrpR0paKumh/HNYIc88SWslrZE0uZB+sqT78rYrJCmn\nD5F0U05fIWlsf5+nmZn1rKwrol3AnIgYD0wEzpc0HpgLLIuIccCyvE7eNh04EZgCXClpUC7rKuBD\nwLj8mpLTZwFPR8QJwJeAy/vjxMzMbO+UEogiYnNE/DQvbwceBEYCU4GFebeFwLS8PBW4MSJ2RsQ6\nYC1wiqRjgMMiYnlEBHBdVZ5KWYuBSZWrJTMzGzhKv0eUu8zeCKwARkTE5rzpMWBEXh4JbChk25jT\nRubl6vQ98kTELmAbcFTDT8DMzPqk1Ef8SDoE+A7w0Yh4tnjBEhEhKfqhDrOB2QDDhw+no6MDSI+4\naWUjhjamDSrtuS/bsWPHfnEejeC26OS2aJzSApGkA0hB6JsRcXNOflzSMRGxOXe7bcnpm4DRheyj\nctqmvFydXsyzUdJg4HDgqep6RMQCYAFAW1tbtLe3AzTkOWv7sjkn7eIL9/X947H+3Pa+V6ZkHR0d\nVD4Xrc5t0clt0ThljZoTcDXwYER8sbBpCTAjL88AbimkT88j4Y4nDUpYmbvxnpU0MZf5vqo8lbLO\nBO7M95HMzGwAKeuK6A+A9wL3Sbonp/01cBmwSNIs4BHgbICIWC1pEfAAacTd+RGxO+c7D7gWGArc\nll+QAt31ktYCW0mj7szMbIApJRBFxI+ArkawTeoiz3xgfo30VcCEGukvAGf1oZpmZtYPSh81Z2Zm\nrc2ByMzMSuVAZGZmpXIgMjOzUjkQmZlZqRyIzMysVA5EZmZWKgciMzMrlQORmZmVyoHIzMxKVeo0\nELb/G9vgp5ivv+z0hpZnZuXzFZGZmZXKgcjMzErlQGRmZqVyIDIzs1I5EJmZWakciMzMrFQORGZm\nVioHIjMzK5UDkZmZlcqByMzMSuVH/Ng+pdGPDAI/NsisbL4iMjOzUjkQmZlZqRyIzMysVL5HZC2v\np/tOc07axcy9uDfle05me2e/viKSNEXSGklrJc0tuz5mZvab9tsrIkmDgH8B/gTYCPxE0pKIeKDc\nmtn+ziP7zPbOfhuIgFOAtRHxMICkG4GpgAOR7XOaEdx6o7tuSgdL6y1FRNl1aApJZwJTIuKDef29\nwKkR8ZGq/WYDs/PqBOD+fq3owHU08GTZlRgg3Bad3Bad3Bad2iLi0N5m3p+viOoSEQuABQCSVkXE\nm0uu0oDgtujktujktujktugkaVVf8u/PgxU2AaML66NympmZDSD7cyD6CTBO0vGSDgSmA0tKrpOZ\nmVXZb7vmImKXpI8AtwODgGsiYnUP2RY0v2b7DLdFJ7dFJ7dFJ7dFpz61xX47WMHMzPYN+3PXnJmZ\n7QMciMzMrFQORLT2o4AkjZb0A0kPSFot6aKcfqSkpZIeyj+HlV3X/iJpkKSfSfpeXm/JtpB0hKTF\nkn4h6UFJb2nhtvhY/v24X9INkg5qlbaQdI2kLZLuL6R1ee6S5uXv0jWSJtdzjJYPRIVHAb0DGA/8\nuaTx5daqX+0C5kTEeGAicH4+/7nAsogYByzL663iIuDBwnqrtsWXge9HxO8Crye1Scu1haSRwIXA\nmyNiAmnw03Rapy2uBaZUpdU89/zdMR04Mee5Mn/HdqvlAxGFRwFFxItA5VFALSEiNkfET/PydtKX\nzUhSGyzMuy0EppVTw/4laRRwOvC1QnLLtYWkw4G3AVcDRMSLEfEMLdgW2WBgqKTBwKuBR2mRtoiI\nu4CtVcldnftU4MaI2BkR64C1pO/YbjkQpS/dDYX1jTmt5UgaC7wRWAGMiIjNedNjwIiSqtXf/hH4\nBPByIa0V2+J44Ang67mb8muSDqYF2yIiNgH/APwK2Axsi4g7aMG2KOjq3Hv1fepAZABIOgT4DvDR\niHi2uC2q+o4BAAAEXUlEQVTSGP/9fpy/pDOALRFxd1f7tEpbkK4A3gRcFRFvBJ6jquupVdoi3/+Y\nSgrOxwIHS3pPcZ9WaYtaGnHuDkR+FBCSDiAFoW9GxM05+XFJx+TtxwBbyqpfP/oD4N2S1pO6aP9Y\n0jdozbbYCGyMiBV5fTEpMLViW7wdWBcRT0TES8DNwFtpzbao6Orce/V96kDU4o8CkiTSfYAHI+KL\nhU1LgBl5eQZwS3/Xrb9FxLyIGBURY0mfgzsj4j20Zls8BmyQ1JaTJpGmUGm5tiB1yU2U9Or8+zKJ\ndC+1FduioqtzXwJMlzRE0vHAOGBlT4X5yQqApHeS7g1UHgU0v+Qq9RtJ/wP4T+A+Ou+L/DXpPtEi\n4DjgEeDsiKi+YbnfktQOfDwizpB0FC3YFpLeQBq0cSDwMPB+0h+vrdgWnwbOIY0y/RnwQeAQWqAt\nJN0AtJOmvXgcuAT4v3Rx7pIuBj5AaquPRsRtPR7DgcjMzMrkrjkzMyuVA5GZmZXKgcjMzErlQGRm\nZqVyIDIzs1I5EJmZWakciMwaQNIb8v+jlVmH9srUFQ0ud1rxifSSOiS9udHHsdblQGTWGG8ASg1E\nTTSNNEWKWVM4EFnLk3SwpFsl3ZsnPjtH0smSfijpbkm3F56r1SHpckkrJf1S0h/mR0N9BjhH0j05\n/8F5QrGV+enVU3P+mZJulvT9PKnY3xfqMUXST3M9lhXq9hvl1HlOe3v8WfmcVkr6qqR/lvRW4N3A\n5/O5vSbvflaxDRrwNlgLG1x2BcwGgCnAoxFxOrwyF89twNSIeELSOcB80mNLAAZHxCm5K+6SiHi7\npL8hTZz2kVzG35KeVfcBSUcAKyX9R87/BtJ0GzuBNZL+CXgB+CrwtohYJ+nIvO/FtcqJiOd6OKea\n+bo5/m7gU6QHm24H7gTujYgfS1oCfC8iFudz+402ID0Y1KxXHIjM0nP2viDpcuB7wNPABGBp/tId\nRJqHpqLyhPK7gbFdlHka6UneH8/rB5GeywVpZsttAJIeAMYAw4C78mRiFJ5Z1lU5xRlkG3H8o4Ef\nFp4X9m3gtd2UX08bmNXFgchaXkT8UtKbSPd4Pke6GlgdEW/pIsvO/HM3Xf8OCfiziFizR6J0aiF/\nT2V0WU4dGnX8rtTTBmZ18T0ia3mSjgWej4hvAJ8HTgWGS3pL3n6ApBN7KGY7cGhh/XbggjxtAJLe\n2EP+5cDb8qPzKXTN7W05vT3+T4D/KWmY0nTYf1bYVn1uZg3lQGQGJ5HuodxDut/xN8CZwOWS7gXu\nIU2E1p0fAOMrgxWAzwIHAD+XtDqvdykingBmAzfnY96UN+1VOQV7e/xNwN+S5o75L2A9sC1vvhH4\nyzzo4TW1SzDrPU8DYWZAmi4+InbkK6Lvkubm+m7Z9bL9n6+IzKzi0nxVeD+wjjT5mVnT+YrIbB8j\naTJweVXyuoj40zLqY9ZXDkRmZlYqd82ZmVmpHIjMzKxUDkRmZlYqByIzMyvV/weP8kz9JLX+ngAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe125ceded0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 句子长度的分布\n",
    "import matplotlib.pyplot as plt\n",
    "df_data['sentence_len'].hist(bins=100)\n",
    "plt.xlim(0, 100)\n",
    "plt.xlabel('sentence_length')\n",
    "plt.ylabel('sentence_num')\n",
    "plt.title('Distribution of the Length of Sentence')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图可以看到，在使用标点进行分割后，绝大部分的句子长度小于30个字。因为一般情况下，我们训练网络的时候都喜欢把输入 padding 到固定的长度，这样子计算更快。因此我们取 32 作为句子长度，超过 32 个字的将把多余的字去掉，少于 32 个字的将用特殊字符填充。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 1.用 chain(*lists) 函数把多个list拼接起来\n",
    "from itertools import chain\n",
    "all_words = list(chain(*df_data['words'].values))\n",
    "# 2.统计所有 word\n",
    "sr_allwords = pd.Series(all_words)\n",
    "sr_allwords = sr_allwords.value_counts()\n",
    "set_words = sr_allwords.index\n",
    "set_ids = range(1, len(set_words)+1) # 注意从1开始，因为我们准备把0作为填充值\n",
    "tags = [ 'x', 's', 'b', 'm', 'e']\n",
    "tag_ids = range(len(tags))\n",
    "\n",
    "# 3. 构建 words 和 tags 都转为数值 id 的映射（使用 Series 比 dict 更加方便）\n",
    "word2id = pd.Series(set_ids, index=set_words)\n",
    "id2word = pd.Series(set_words, index=set_ids)\n",
    "tag2id = pd.Series(tag_ids, index=tags)\n",
    "id2tag = pd.Series(tags, index=tag_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_size=5158\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(set_words)\n",
    "print 'vocab_size={}'.format(vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把 words 和 tags 都转为数值 id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 21s, sys: 164 ms, total: 2min 21s\n",
      "Wall time: 2min 21s\n",
      "CPU times: user 2min 20s, sys: 364 ms, total: 2min 20s\n",
      "Wall time: 2min 20s\n"
     ]
    }
   ],
   "source": [
    "max_len = 32\n",
    "def X_padding(words):\n",
    "    \"\"\"把 words 转为 id 形式，并自动补全位 max_len 长度。\"\"\"\n",
    "    ids = list(word2id[words])\n",
    "    if len(ids) >= max_len:  # 长则弃掉\n",
    "        return ids[:max_len]\n",
    "    ids.extend([0]*(max_len-len(ids))) # 短则补全\n",
    "    return ids\n",
    "\n",
    "def y_padding(tags):\n",
    "    \"\"\"把 tags 转为 id 形式， 并自动补全位 max_len 长度。\"\"\"\n",
    "    ids = list(tag2id[tags])\n",
    "    if len(ids) >= max_len:  # 长则弃掉\n",
    "        return ids[:max_len]\n",
    "    ids.extend([0]*(max_len-len(ids))) # 短则补全\n",
    "    return ids\n",
    "\n",
    "%time df_data['X'] = df_data['words'].apply(X_padding)\n",
    "%time df_data['y'] = df_data['tags'].apply(y_padding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape=(321827, 32), y.shape=(321827, 32)\n",
      "Example of words:  [u'\\u4eba' u'\\u4eec' u'\\u5e38' u'\\u8bf4' u'\\u751f' u'\\u6d3b' u'\\u662f'\n",
      " u'\\u4e00' u'\\u90e8' u'\\u6559' u'\\u79d1' u'\\u4e66']\n",
      "Example of X:  [  8  43 320  88  36 198   7   2  41 163 124 245   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0]\n",
      "Example of tags:  [u'b' u'e' u's' u's' u'b' u'e' u's' u's' u's' u'b' u'm' u'e']\n",
      "Example of y:  [2 4 1 1 2 4 1 1 1 2 3 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "# 最后得到了所有的数据\n",
    "X = np.asarray(list(df_data['X'].values))\n",
    "y = np.asarray(list(df_data['y'].values))\n",
    "print 'X.shape={}, y.shape={}'.format(X.shape, y.shape)\n",
    "print 'Example of words: ', df_data['words'].values[0]\n",
    "print 'Example of X: ', X[0]\n",
    "print 'Example of tags: ', df_data['tags'].values[0]\n",
    "print 'Example of y: ', y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.67 s, sys: 216 ms, total: 6.88 s\n",
      "Wall time: 6.88 s\n",
      "CPU times: user 6.67 s, sys: 208 ms, total: 6.88 s\n",
      "Wall time: 6.87 s\n",
      "** Finished saving the data.\n"
     ]
    }
   ],
   "source": [
    "# 保存数据\n",
    "import pickle\n",
    "import os\n",
    "\n",
    "if not os.path.exists('data/'):\n",
    "    os.makedirs('data/')\n",
    "\n",
    "with open('data/data.pkl', 'wb') as outp:\n",
    "    %time pickle.dump(X, outp)\n",
    "    %time pickle.dump(y, outp)\n",
    "    pickle.dump(word2id, outp)\n",
    "    pickle.dump(id2word, outp)\n",
    "    pickle.dump(tag2id, outp)\n",
    "    pickle.dump(id2tag, outp)\n",
    "print '** Finished saving the data.'    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.58 s, sys: 216 ms, total: 1.8 s\n",
      "Wall time: 1.79 s\n",
      "CPU times: user 1.4 s, sys: 156 ms, total: 1.56 s\n",
      "Wall time: 1.55 s\n",
      "X_train.shape=(205968, 32), y_train.shape=(205968, 32); \n",
      "X_valid.shape=(51493, 32), y_valid.shape=(51493, 32);\n",
      "X_test.shape=(64366, 32), y_test.shape=(64366, 32)\n"
     ]
    }
   ],
   "source": [
    "# 导入数据\n",
    "import pickle\n",
    "with open('data/data.pkl', 'rb') as inp:\n",
    "    %time X = pickle.load(inp)\n",
    "    %time y = pickle.load(inp)\n",
    "    word2id = pickle.load(inp)\n",
    "    id2word = pickle.load(inp)\n",
    "    tag2id = pickle.load(inp)\n",
    "    id2tag = pickle.load(inp)\n",
    "\n",
    "# 划分测试集/训练集/验证集\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train,  test_size=0.2, random_state=42)\n",
    "print 'X_train.shape={}, y_train.shape={}; \\nX_valid.shape={}, y_valid.shape={};\\nX_test.shape={}, y_test.shape={}'.format(\n",
    "    X_train.shape, y_train.shape, X_valid.shape, y_valid.shape, X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 构造一个生成batch数据的类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating the data generator ...\n",
      "Finished creating the data generator.\n"
     ]
    }
   ],
   "source": [
    "# ** 3.build the data generator\n",
    "class BatchGenerator(object):\n",
    "    \"\"\" Construct a Data generator. The input X, y should be ndarray or list like type.\n",
    "    \n",
    "    Example:\n",
    "        Data_train = BatchGenerator(X=X_train_all, y=y_train_all, shuffle=False)\n",
    "        Data_test = BatchGenerator(X=X_test_all, y=y_test_all, shuffle=False)\n",
    "        X = Data_train.X\n",
    "        y = Data_train.y\n",
    "        or:\n",
    "        X_batch, y_batch = Data_train.next_batch(batch_size)\n",
    "     \"\"\" \n",
    "    \n",
    "    def __init__(self, X, y, shuffle=False):\n",
    "        if type(X) != np.ndarray:\n",
    "            X = np.asarray(X)\n",
    "        if type(y) != np.ndarray:\n",
    "            y = np.asarray(y)\n",
    "        self._X = X\n",
    "        self._y = y\n",
    "        self._epochs_completed = 0\n",
    "        self._index_in_epoch = 0\n",
    "        self._number_examples = self._X.shape[0]\n",
    "        self._shuffle = shuffle\n",
    "        if self._shuffle:\n",
    "            new_index = np.random.permutation(self._number_examples)\n",
    "            self._X = self._X[new_index]\n",
    "            self._y = self._y[new_index]\n",
    "                \n",
    "    @property\n",
    "    def X(self):\n",
    "        return self._X\n",
    "    \n",
    "    @property\n",
    "    def y(self):\n",
    "        return self._y\n",
    "    \n",
    "    @property\n",
    "    def num_examples(self):\n",
    "        return self._number_examples\n",
    "    \n",
    "    @property\n",
    "    def epochs_completed(self):\n",
    "        return self._epochs_completed\n",
    "    \n",
    "    def next_batch(self, batch_size):\n",
    "        \"\"\" Return the next 'batch_size' examples from this data set.\"\"\"\n",
    "        start = self._index_in_epoch\n",
    "        self._index_in_epoch += batch_size\n",
    "        if self._index_in_epoch > self._number_examples:\n",
    "            # finished epoch\n",
    "            self._epochs_completed += 1\n",
    "            # Shuffle the data \n",
    "            if self._shuffle:\n",
    "                new_index = np.random.permutation(self._number_examples)\n",
    "                self._X = self._X[new_index]\n",
    "                self._y = self._y[new_index]\n",
    "            start = 0\n",
    "            self._index_in_epoch = batch_size\n",
    "            assert batch_size <= self._number_examples\n",
    "        end = self._index_in_epoch\n",
    "        return self._X[start:end], self._y[start:end]\n",
    "\n",
    "print 'Creating the data generator ...'\n",
    "data_train = BatchGenerator(X_train, y_train, shuffle=True)\n",
    "data_valid = BatchGenerator(X_valid, y_valid, shuffle=False)\n",
    "data_test = BatchGenerator(X_test, y_test, shuffle=False)\n",
    "print 'Finished creating the data generator.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Bi-directional lstm 模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 模型构造"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished creating the bi-lstm model.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth = True\n",
    "sess = tf.Session(config=config)\n",
    "from tensorflow.contrib import rnn\n",
    "import numpy as np\n",
    "\n",
    "'''\n",
    "For Chinese word segmentation.\n",
    "'''\n",
    "# ##################### config ######################\n",
    "decay = 0.85\n",
    "max_epoch = 5\n",
    "max_max_epoch = 10\n",
    "timestep_size = max_len = 32           # 句子长度\n",
    "vocab_size = 5159    # 样本中不同字的个数+1(padding 0)，根据处理数据的时候得到\n",
    "input_size = embedding_size = 64       # 字向量长度\n",
    "class_num = 5\n",
    "hidden_size = 128    # 隐含层节点数\n",
    "layer_num = 2        # bi-lstm 层数\n",
    "max_grad_norm = 5.0  # 最大梯度（超过此值的梯度将被裁剪）\n",
    "\n",
    "lr = tf.placeholder(tf.float32, [])\n",
    "keep_prob = tf.placeholder(tf.float32, [])\n",
    "batch_size = tf.placeholder(tf.int32, [])  # 注意类型必须为 tf.int32\n",
    "model_save_path = 'ckpt/bi-lstm.ckpt'  # 模型保存位置\n",
    "\n",
    "\n",
    "with tf.variable_scope('embedding'):\n",
    "    embedding = tf.get_variable(\"embedding\", [vocab_size, embedding_size], dtype=tf.float32)\n",
    "\n",
    "def weight_variable(shape):\n",
    "    \"\"\"Create a weight variable with appropriate initialization.\"\"\"\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "    \"\"\"Create a bias variable with appropriate initialization.\"\"\"\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def lstm_cell():\n",
    "    cell = rnn.LSTMCell(hidden_size, reuse=tf.get_variable_scope().reuse)\n",
    "    return rnn.DropoutWrapper(cell, output_keep_prob=keep_prob)\n",
    "         \n",
    "def bi_lstm(X_inputs):\n",
    "    \"\"\"build the bi-LSTMs network. Return the y_pred\"\"\"\n",
    "    # X_inputs.shape = [batchsize, timestep_size]  ->  inputs.shape = [batchsize, timestep_size, embedding_size]\n",
    "    inputs = tf.nn.embedding_lookup(embedding, X_inputs)  \n",
    "    \n",
    "    # ** 1.构建前向后向多层 LSTM\n",
    "    cell_fw = rnn.MultiRNNCell([lstm_cell() for _ in range(layer_num)], state_is_tuple=True)\n",
    "    cell_bw = rnn.MultiRNNCell([lstm_cell() for _ in range(layer_num)], state_is_tuple=True)\n",
    "  \n",
    "    # ** 2.初始状态\n",
    "    initial_state_fw = cell_fw.zero_state(batch_size, tf.float32)\n",
    "    initial_state_bw = cell_bw.zero_state(batch_size, tf.float32)  \n",
    "    \n",
    "    # 下面两部分是等价的\n",
    "    # **************************************************************\n",
    "    # ** 把 inputs 处理成 rnn.static_bidirectional_rnn 的要求形式\n",
    "    # ** 文档说明\n",
    "    # inputs: A length T list of inputs, each a tensor of shape\n",
    "    # [batch_size, input_size], or a nested tuple of such elements.\n",
    "    # *************************************************************\n",
    "    # Unstack to get a list of 'n_steps' tensors of shape (batch_size, n_input)\n",
    "    # inputs.shape = [batchsize, timestep_size, embedding_size]  ->  timestep_size tensor, each_tensor.shape = [batchsize, embedding_size]\n",
    "    # inputs = tf.unstack(inputs, timestep_size, 1)\n",
    "    # ** 3.bi-lstm 计算（tf封装）  一般采用下面 static_bidirectional_rnn 函数调用。\n",
    "    #   但是为了理解计算的细节，所以把后面的这段代码进行展开自己实现了一遍。\n",
    "#     try:\n",
    "#         outputs, _, _ = rnn.static_bidirectional_rnn(cell_fw, cell_bw, inputs, \n",
    "#                         initial_state_fw = initial_state_fw, initial_state_bw = initial_state_bw, dtype=tf.float32)\n",
    "#     except Exception: # Old TensorFlow version only returns outputs not states\n",
    "#         outputs = rnn.static_bidirectional_rnn(cell_fw, cell_bw, inputs, \n",
    "#                         initial_state_fw = initial_state_fw, initial_state_bw = initial_state_bw, dtype=tf.float32)\n",
    "#     output = tf.reshape(tf.concat(outputs, 1), [-1, hidden_size * 2])\n",
    "    # ***********************************************************\n",
    "    \n",
    "    # ***********************************************************\n",
    "    # ** 3. bi-lstm 计算（展开）\n",
    "    with tf.variable_scope('bidirectional_rnn'):\n",
    "        # *** 下面，两个网络是分别计算 output 和 state \n",
    "        # Forward direction\n",
    "        outputs_fw = list()\n",
    "        state_fw = initial_state_fw\n",
    "        with tf.variable_scope('fw'):\n",
    "            for timestep in range(timestep_size):\n",
    "                if timestep > 0:\n",
    "                    tf.get_variable_scope().reuse_variables()\n",
    "                (output_fw, state_fw) = cell_fw(inputs[:, timestep, :], state_fw)\n",
    "                outputs_fw.append(output_fw)\n",
    "        \n",
    "        # backward direction\n",
    "        outputs_bw = list()\n",
    "        state_bw = initial_state_bw\n",
    "        with tf.variable_scope('bw') as bw_scope:\n",
    "            inputs = tf.reverse(inputs, [1])\n",
    "            for timestep in range(timestep_size):\n",
    "                if timestep > 0:\n",
    "                    tf.get_variable_scope().reuse_variables()\n",
    "                (output_bw, state_bw) = cell_bw(inputs[:, timestep, :], state_bw)\n",
    "                outputs_bw.append(output_bw)\n",
    "        # *** 然后把 output_bw 在 timestep 维度进行翻转\n",
    "        # outputs_bw.shape = [timestep_size, batch_size, hidden_size]\n",
    "        outputs_bw = tf.reverse(outputs_bw, [0])\n",
    "        # 把两个oupputs 拼成 [timestep_size, batch_size, hidden_size*2]\n",
    "        output = tf.concat([outputs_fw, outputs_bw], 2)\n",
    "        output = tf.transpose(output, perm=[1,0,2])\n",
    "        output = tf.reshape(output, [-1, hidden_size*2])\n",
    "    # ***********************************************************\n",
    "    return output # [-1, hidden_size*2]\n",
    "\n",
    "\n",
    "with tf.variable_scope('Inputs'):\n",
    "    X_inputs = tf.placeholder(tf.int32, [None, timestep_size], name='X_input')\n",
    "    y_inputs = tf.placeholder(tf.int32, [None, timestep_size], name='y_input')   \n",
    "    \n",
    "bilstm_output = bi_lstm(X_inputs)\n",
    "\n",
    "with tf.variable_scope('outputs'):\n",
    "    softmax_w = weight_variable([hidden_size * 2, class_num]) \n",
    "    softmax_b = bias_variable([class_num]) \n",
    "    y_pred = tf.matmul(bilstm_output, softmax_w) + softmax_b\n",
    "\n",
    "# adding extra statistics to monitor\n",
    "# y_inputs.shape = [batch_size, timestep_size]\n",
    "correct_prediction = tf.equal(tf.cast(tf.argmax(y_pred, 1), tf.int32), tf.reshape(y_inputs, [-1]))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(labels = tf.reshape(y_inputs, [-1]), logits = y_pred))\n",
    "\n",
    "# ***** 优化求解 *******\n",
    "tvars = tf.trainable_variables()  # 获取模型的所有参数\n",
    "grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), max_grad_norm)  # 获取损失函数对于每个参数的梯度\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=lr)   # 优化器\n",
    "\n",
    "# 梯度下降计算\n",
    "train_op = optimizer.apply_gradients( zip(grads, tvars),\n",
    "    global_step=tf.contrib.framework.get_or_create_global_step())\n",
    "print 'Finished creating the bi-lstm model.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EPOCH 1， lr=0.0001\n",
      "\ttraining acc=0.705096, cost=0.778346;  valid acc= 0.75491, cost=0.536262 \n",
      "\ttraining acc=0.757477, cost=0.519743;  valid acc= 0.771731, cost=0.478449 \n",
      "\ttraining acc=0.775235, cost=0.475189;  valid acc= 0.789378, cost=0.451933 \n",
      "\ttraining acc=0.788222, cost=0.448686;  valid acc= 0.793968, cost=0.431898 \n",
      "\ttraining acc=0.795513, cost=0.432824;  valid acc= 0.80192, cost=0.417867 \n",
      "\ttraining 205968, acc=0.764414, cost=0.530666 \n",
      "Epoch training 205968, acc=0.764414, cost=0.530666, speed=142.158 s/epoch\n",
      "EPOCH 2， lr=0.0001\n",
      "\ttraining acc=0.803306, cost=0.417545;  valid acc= 0.808482, cost=0.403548 \n",
      "\ttraining acc=0.809051, cost=0.403985;  valid acc= 0.817827, cost=0.386311 \n",
      "\ttraining acc=0.83582, cost=0.369938;  valid acc= 0.864623, cost=0.332383 \n",
      "\ttraining acc=0.870974, cost=0.320522;  valid acc= 0.889104, cost=0.286238 \n",
      "\ttraining acc=0.889264, cost=0.289025;  valid acc= 0.899717, cost=0.268476 \n",
      "\ttraining 205968, acc=0.841816, cost=0.359989 \n",
      "Epoch training 205968, acc=0.841816, cost=0.359989, speed=143.833 s/epoch\n",
      "EPOCH 3， lr=0.0001\n",
      "\ttraining acc=0.897466, cost=0.270918;  valid acc= 0.904032, cost=0.254307 \n",
      "\ttraining acc=0.902237, cost=0.261452;  valid acc= 0.908694, cost=0.24403 \n",
      "\ttraining acc=0.906363, cost=0.250787;  valid acc= 0.91161, cost=0.236932 \n",
      "\ttraining acc=0.909716, cost=0.242962;  valid acc= 0.914322, cost=0.230206 \n",
      "\ttraining acc=0.911909, cost=0.237687;  valid acc= 0.917235, cost=0.223029 \n",
      "the save path is  ckpt/bi-lstm.ckpt-3\n",
      "\ttraining 205968, acc=0.905567, cost=0.252686 \n",
      "Epoch training 205968, acc=0.905567, cost=0.252686, speed=142.899 s/epoch\n",
      "EPOCH 4， lr=0.0001\n",
      "\ttraining acc=0.915506, cost=0.228171;  valid acc= 0.919788, cost=0.216495 \n",
      "\ttraining acc=0.917144, cost=0.223763;  valid acc= 0.922572, cost=0.208754 \n",
      "\ttraining acc=0.921049, cost=0.213782;  valid acc= 0.925303, cost=0.201429 \n",
      "\ttraining acc=0.923136, cost=0.207814;  valid acc= 0.927573, cost=0.194483 \n",
      "\ttraining acc=0.926754, cost=0.198604;  valid acc= 0.930409, cost=0.188282 \n",
      "\ttraining 205968, acc=0.920724, cost=0.214414 \n",
      "Epoch training 205968, acc=0.920724, cost=0.214414, speed=141.494 s/epoch\n",
      "EPOCH 5， lr=0.0001\n",
      "\ttraining acc=0.928721, cost=0.193629;  valid acc= 0.932203, cost=0.182827 \n",
      "\ttraining acc=0.930034, cost=0.190198;  valid acc= 0.934062, cost=0.178452 \n",
      "\ttraining acc=0.932533, cost=0.183748;  valid acc= 0.935541, cost=0.175836 \n",
      "\ttraining acc=0.933429, cost=0.181398;  valid acc= 0.93667, cost=0.171729 \n",
      "\ttraining acc=0.933569, cost=0.180416;  valid acc= 0.937609, cost=0.168493 \n",
      "\ttraining 205968, acc=0.93167, cost=0.185841 \n",
      "Epoch training 205968, acc=0.93167, cost=0.185841, speed=142.189 s/epoch\n",
      "EPOCH 6， lr=0.0001\n",
      "\ttraining acc=0.936004, cost=0.173919;  valid acc= 0.938591, cost=0.166248 \n",
      "\ttraining acc=0.93659, cost=0.172726;  valid acc= 0.939384, cost=0.163917 \n",
      "\ttraining acc=0.9376, cost=0.170443;  valid acc= 0.940012, cost=0.162237 \n",
      "\ttraining acc=0.937876, cost=0.169466;  valid acc= 0.94095, cost=0.160298 \n",
      "\ttraining acc=0.939148, cost=0.166432;  valid acc= 0.941518, cost=0.158338 \n",
      "the save path is  ckpt/bi-lstm.ckpt-6\n",
      "\ttraining 205968, acc=0.937462, cost=0.170551 \n",
      "Epoch training 205968, acc=0.937462, cost=0.170551, speed=143.199 s/epoch\n",
      "**TEST RESULT:\n",
      "**Test 64366, acc=0.941968, cost=0.157662\n"
     ]
    }
   ],
   "source": [
    "def test_epoch(dataset):\n",
    "    \"\"\"Testing or valid.\"\"\"\n",
    "    _batch_size = 500\n",
    "    fetches = [accuracy, cost]\n",
    "    _y = dataset.y\n",
    "    data_size = _y.shape[0]\n",
    "    batch_num = int(data_size / _batch_size)\n",
    "    start_time = time.time()\n",
    "    _costs = 0.0\n",
    "    _accs = 0.0\n",
    "    for i in xrange(batch_num):\n",
    "        X_batch, y_batch = dataset.next_batch(_batch_size)\n",
    "        feed_dict = {X_inputs:X_batch, y_inputs:y_batch, lr:1e-5, batch_size:_batch_size, keep_prob:1.0}\n",
    "        _acc, _cost = sess.run(fetches, feed_dict)\n",
    "        _accs += _acc\n",
    "        _costs += _cost    \n",
    "    mean_acc= _accs / batch_num     \n",
    "    mean_cost = _costs / batch_num\n",
    "    return mean_acc, mean_cost\n",
    "\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "tr_batch_size = 128 \n",
    "max_max_epoch = 6\n",
    "display_num = 5  # 每个 epoch 显示是个结果\n",
    "tr_batch_num = int(data_train.y.shape[0] / tr_batch_size)  # 每个 epoch 中包含的 batch 数\n",
    "display_batch = int(tr_batch_num / display_num)  # 每训练 display_batch 之后输出一次\n",
    "saver = tf.train.Saver(max_to_keep=10)  # 最多保存的模型数量\n",
    "for epoch in xrange(max_max_epoch):\n",
    "    _lr = 1e-4\n",
    "    if epoch > max_epoch:\n",
    "        _lr = _lr * ((decay) ** (epoch - max_epoch))\n",
    "    print 'EPOCH %d， lr=%g' % (epoch+1, _lr)\n",
    "    start_time = time.time()\n",
    "    _costs = 0.0\n",
    "    _accs = 0.0\n",
    "    show_accs = 0.0\n",
    "    show_costs = 0.0\n",
    "    for batch in xrange(tr_batch_num): \n",
    "        fetches = [accuracy, cost, train_op]\n",
    "        X_batch, y_batch = data_train.next_batch(tr_batch_size)\n",
    "        feed_dict = {X_inputs:X_batch, y_inputs:y_batch, lr:_lr, batch_size:tr_batch_size, keep_prob:0.5}\n",
    "        _acc, _cost, _ = sess.run(fetches, feed_dict) # the cost is the mean cost of one batch\n",
    "        _accs += _acc\n",
    "        _costs += _cost\n",
    "        show_accs += _acc\n",
    "        show_costs += _cost\n",
    "        if (batch + 1) % display_batch == 0:\n",
    "            valid_acc, valid_cost = test_epoch(data_valid)  # valid\n",
    "            print '\\ttraining acc=%g, cost=%g;  valid acc= %g, cost=%g ' % (show_accs / display_batch,\n",
    "                                                show_costs / display_batch, valid_acc, valid_cost)\n",
    "            show_accs = 0.0\n",
    "            show_costs = 0.0\n",
    "    mean_acc = _accs / tr_batch_num \n",
    "    mean_cost = _costs / tr_batch_num\n",
    "    if (epoch + 1) % 3 == 0:  # 每 3 个 epoch 保存一次模型\n",
    "        save_path = saver.save(sess, model_save_path, global_step=(epoch+1))\n",
    "        print 'the save path is ', save_path\n",
    "    print '\\ttraining %d, acc=%g, cost=%g ' % (data_train.y.shape[0], mean_acc, mean_cost)\n",
    "    print 'Epoch training %d, acc=%g, cost=%g, speed=%g s/epoch' % (data_train.y.shape[0], mean_acc, mean_cost, time.time()-start_time)        \n",
    "# testing\n",
    "print '**TEST RESULT:'\n",
    "test_acc, test_cost = test_epoch(data_test)\n",
    "print '**Test %d, acc=%g, cost=%g' % (data_test.y.shape[0], test_acc, test_cost) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型测试，现在给定一个字符串，首先应该把处理成正确的模型输入形式。即每次输入一个片段，（这里限制了每个片段的长度不超过 max_len=32）。每个字处理为对应的 id， 每个片段都会 padding 处理到固定的长度。也就是说，输入的是一个list， list 的每个元素是一个包含多个 id 的list。<br/>\n",
    "即 [[id0, id1, ..., id31], [id0, id1, ..., id31], [], ...]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ckpt/bi-lstm.ckpt-6\n",
      "CPU times: user 444 ms, sys: 148 ms, total: 592 ms\n",
      "Wall time: 401 ms\n"
     ]
    }
   ],
   "source": [
    "# ** 导入模型\n",
    "saver = tf.train.Saver()\n",
    "best_model_path = 'ckpt/bi-lstm.ckpt-6'\n",
    "%time saver.restore(sess, best_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_tt.shape= (2, 32) y_tt.shape= (2, 32)\n",
      "X_tt =  [[1690  378  308  735  311   28  490   18    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0]\n",
      " [   2  368   57  283  941 1239  252  191    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0]]\n",
      "y_tt =  [[1 1 2 4 2 4 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [1 1 2 4 2 4 2 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# 再看看模型的输入数据形式, 我们要进行分词，首先就要把句子转为这样的形式\n",
    "X_tt, y_tt = data_train.next_batch(2)\n",
    "print 'X_tt.shape=', X_tt.shape, 'y_tt.shape=', y_tt.shape\n",
    "print 'X_tt = ', X_tt\n",
    "print 'y_tt = ', y_tt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 维特比解码\n",
    "下面使用维特比的方法来计算的最优的状态序列。具体原理可以参考 隐马尔可夫模型。在 HMM 监督学习中，我们统计样本频数来得到转移概率和 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the transition probability: \n",
      "be 0.828739514282\n",
      "bm 0.171260485718\n",
      "eb 0.592369623847\n",
      "es 0.407630376153\n",
      "me 0.504871829789\n",
      "mm 0.495128170211\n",
      "sb 0.623266273388\n",
      "ss 0.376733726612\n"
     ]
    }
   ],
   "source": [
    "# 利用 labels（即状态序列）来统计转移概率\n",
    "# 因为状态数比较少，这里用 dict={'I_tI_{t+1}'：p} 来实现\n",
    "# A统计状态转移的频数\n",
    "A = {\n",
    "      'sb':0,\n",
    "      'ss':0,\n",
    "      'be':0,\n",
    "      'bm':0,\n",
    "      'me':0,\n",
    "      'mm':0,\n",
    "      'eb':0,\n",
    "      'es':0\n",
    "     }\n",
    "\n",
    "# zy 表示转移概率矩阵\n",
    "zy = dict()\n",
    "for label in labels:\n",
    "    for t in xrange(len(label) - 1):\n",
    "        key = label[t] + label[t+1]\n",
    "        A[key] += 1.0\n",
    "        \n",
    "zy['sb'] = A['sb'] / (A['sb'] + A['ss'])\n",
    "zy['ss'] = 1.0 - zy['sb']\n",
    "zy['be'] = A['be'] / (A['be'] + A['bm'])\n",
    "zy['bm'] = 1.0 - zy['be']\n",
    "zy['me'] = A['me'] / (A['me'] + A['mm'])\n",
    "zy['mm'] = 1.0 - zy['me']\n",
    "zy['eb'] = A['eb'] / (A['eb'] + A['es'])\n",
    "zy['es'] = 1.0 - zy['eb']\n",
    "keys = sorted(zy.keys())\n",
    "print 'the transition probability: '\n",
    "for key in keys:\n",
    "    print key, zy[key]\n",
    "    \n",
    "zy = {i:np.log(zy[i]) for i in zy.keys()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def viterbi(nodes):\n",
    "    \"\"\"\n",
    "    维特比译码：除了第一层以外，每一层有4个节点。\n",
    "    计算当前层（第一层不需要计算）四个节点的最短路径：\n",
    "       对于本层的每一个节点，计算出路径来自上一层的各个节点的新的路径长度（概率）。保留最大值（最短路径）。\n",
    "       上一层每个节点的路径保存在 paths 中。计算本层的时候，先用paths_ 暂存，然后把本层的最大路径保存到 paths 中。\n",
    "       paths 采用字典的形式保存（路径：路径长度）。\n",
    "       一直计算到最后一层，得到四条路径，将长度最短（概率值最大的路径返回）\n",
    "    \"\"\"\n",
    "    paths = {'b': nodes[0]['b'], 's':nodes[0]['s']} # 第一层，只有两个节点\n",
    "    for layer in xrange(1, len(nodes)):  # 后面的每一层\n",
    "        paths_ = paths.copy()  # 先保存上一层的路径\n",
    "        # node_now 为本层节点， node_last 为上层节点\n",
    "        paths = {}  # 清空 path \n",
    "        for node_now in nodes[layer].keys():\n",
    "            # 对于本层的每个节点，找出最短路径\n",
    "            sub_paths = {} \n",
    "            # 上一层的每个节点到本层节点的连接\n",
    "            for path_last in paths_.keys():\n",
    "                if path_last[-1] + node_now in zy.keys(): # 若转移概率不为 0 \n",
    "                    sub_paths[path_last + node_now] = paths_[path_last] + nodes[layer][node_now] + zy[path_last[-1] + node_now]\n",
    "            # 最短路径,即概率最大的那个\n",
    "            sr_subpaths = pd.Series(sub_paths)\n",
    "            sr_subpaths = sr_subpaths.sort_values()  # 升序排序\n",
    "            node_subpath = sr_subpaths.index[-1]  # 最短路径\n",
    "            node_value = sr_subpaths[-1]   # 最短路径对应的值\n",
    "            # 把 node_now 的最短路径添加到 paths 中\n",
    "            paths[node_subpath] = node_value\n",
    "    # 所有层求完后，找出最后一层中各个节点的路径最短的路径\n",
    "    sr_paths = pd.Series(paths)\n",
    "    sr_paths = sr_paths.sort_values()  # 按照升序排序\n",
    "    return sr_paths.index[-1]  # 返回最短路径（概率值最大的路径）\n",
    "\n",
    "\n",
    "def text2ids(text):\n",
    "    \"\"\"把字片段text转为 ids.\"\"\"\n",
    "    words = list(text)\n",
    "    ids = list(word2id[words])\n",
    "    if len(ids) >= max_len:  # 长则弃掉\n",
    "        print u'输出片段超过%d部分无法处理' % (max_len) \n",
    "        return ids[:max_len]\n",
    "    ids.extend([0]*(max_len-len(ids))) # 短则补全\n",
    "    ids = np.asarray(ids).reshape([-1, max_len])\n",
    "    return ids\n",
    "\n",
    "\n",
    "def simple_cut(text):\n",
    "    \"\"\"对一个片段text（标点符号把句子划分为多个片段）进行预测。\"\"\"\n",
    "    if text:\n",
    "        text_len = len(text)\n",
    "        X_batch = text2ids(text)  # 这里每个 batch 是一个样本\n",
    "        fetches = [y_pred]\n",
    "        feed_dict = {X_inputs:X_batch, lr:1.0, batch_size:1, keep_prob:1.0}\n",
    "        _y_pred = sess.run(fetches, feed_dict)[0][:text_len]  # padding填充的部分直接丢弃\n",
    "        nodes = [dict(zip(['s','b','m','e'], each[1:])) for each in _y_pred]\n",
    "        tags = viterbi(nodes)\n",
    "        words = []\n",
    "        for i in range(len(text)):\n",
    "            if tags[i] in ['s', 'b']:\n",
    "                words.append(text[i])\n",
    "            else:\n",
    "                words[-1] += text[i]\n",
    "        return words\n",
    "    else:\n",
    "        return []\n",
    "\n",
    "\n",
    "def cut_word(sentence):\n",
    "    \"\"\"首先将一个sentence根据标点和英文符号/字符串划分成多个片段text，然后对每一个片段分词。\"\"\"\n",
    "    not_cuts = re.compile(u'([0-9\\da-zA-Z ]+)|[。，、？！.\\.\\?,!]')\n",
    "    result = []\n",
    "    start = 0\n",
    "    for seg_sign in not_cuts.finditer(sentence):\n",
    "        result.extend(simple_cut(sentence[start:seg_sign.start()]))\n",
    "        result.append(sentence[seg_sign.start():seg_sign.end()])\n",
    "        start = seg_sign.end()\n",
    "    result.extend(simple_cut(sentence[start:]))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "人们 / 思考 / 问题 / 往往 / 不是 / 从 / 零开始 / 的 / 。 / 就 / 好像 / 你 / 现在 / 阅读 / 这篇 / 文章 / 一样 / ， / 你 / 对 / 每个 / 词 / 的 / 理解 / 都会 / 依赖 / 于 / 你 / 前面 / 看到 / 的 / 一些 / 词 / ， /        / 而 / 不是 / 把 / 你 / 前面 / 看 / 的 / 内容 / 全部 / 抛弃 / 了 / ， / 忘记 / 了 / ， / 再去 / 理解 / 这个 / 单词 / 。 / 也 / 就是 / 说 / ， / 人们 / 的 / 思维 / 总是 / 会有 / 延续 / 性 / 的 / 。 / \n"
     ]
    }
   ],
   "source": [
    "# 例一\n",
    "sentence = u'人们思考问题往往不是从零开始的。就好像你现在阅读这篇文章一样，你对每个词的理解都会依赖于你前面看到的一些词，\\\n",
    "      而不是把你前面看的内容全部抛弃了，忘记了，再去理解这个单词。也就是说，人们的思维总是会有延续性的。'\n",
    "result = cut_word(sentence)\n",
    "rss = ''\n",
    "for each in result:\n",
    "    rss = rss + each + ' / '\n",
    "print rss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "天舟 / 一号 / 是 / 我国 / 自主 / 研制 / 的 / 首艘 / 货运 / 飞船 / ， / 由于 / 它 / 只 / 运货 / ， / 不送 / 人 / ， / 所以 / 被 / 形象 / 地 / 称为 / 太空 / “快 / 递小 / 哥” / 。 /      / 它 / 采用 / 两舱 / 式 / 结构 / ， / 直径 / 较小 / 的 / 是 / 推进 / 舱 / ， / 直径 / 较大 / 的 / 为 / 货物 / 舱 / 。 / 其 / 最大 / 直径 / 达到 / 3 / . / 35 / 米 / ， / 飞船 / 全长 / 10 / . / 6 / 米 / ， / 载荷 / 能力 / 达到 / 了 / 6 / . / 5 / 吨 / ， /      / 满载 / 货物 / 时重 / 13 / . / 5 / 吨 / 。 / 如果 / 此次 / 满载 / 的 / 话 / ， / 它 / 很 / 可能 / 将 / 成为 / 中国 / 发射 / 进入 / 太空 / 的 / 质量 / 最大 / 的 / 有效 / 载荷 / 。 / 甚至 / 比天宫 / 二号 / 空间 / 实验 / 室 / 还大 / ， /      / 后者 / 全长 / 10 / . / 4 / 米 / ， / 直径 / 同为 / 3 / . / 35 / 米 / ， / 质量 / 为 / 8 / . / 6 / 吨 / 。 / \n"
     ]
    }
   ],
   "source": [
    "# 例二\n",
    "sentence = u'天舟一号是我国自主研制的首艘货运飞船，由于它只运货，不送人，所以被形象地称为太空“快递小哥”。\\\n",
    "    它采用两舱式结构，直径较小的是推进舱，直径较大的为货物舱。其最大直径达到3.35米，飞船全长10.6米，载荷能力达到了6.5吨，\\\n",
    "    满载货物时重13.5吨。如果此次满载的话，它很可能将成为中国发射进入太空的质量最大的有效载荷。甚至比天宫二号空间实验室还大，\\\n",
    "    后者全长10.4米，直径同为3.35米，质量为8.6吨。'\n",
    "result = cut_word(sentence)\n",
    "rss = ''\n",
    "for each in result:\n",
    "    rss = rss + each + ' / '\n",
    "print rss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "南京/ 市长江大桥/ \n"
     ]
    }
   ],
   "source": [
    "# 例三\n",
    "sentence = u'南京市长江大桥'\n",
    "result = cut_word(sentence)\n",
    "rss = ''\n",
    "for each in result:\n",
    "    rss = rss + each + '/ '\n",
    "print rss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结论：本例子使用 Bi-directional LSTM 来完成了序列标注的问题。本例中展示的是一个分词任务，但是还有其他的序列标注问题都是可以通过这样一个架构来实现的，比如 POS（词性标注）、NER（命名实体识别）等。在本例中，最后的分词效果还不是非常好，但已经达到了实用的水平，而且模型也只是粗略地跑了一遍，还没有进行任何的参数优化。最后的维特比译码中转移概率根据训练语料进行统计。\n",
    "\n",
    "看到最后一个 \"南京/ 市长江大桥/\" 的结果时，心里不由的一阵哀伤...\n",
    "\n",
    "在模型构造中，我们对 Bi-directional LSTM 模型进行了比较详细的展开分析，从而对模型有了深入的理解。这很大程度上也得益于 TensorFlow 比较底层，如果是用 keras 框架的话，虽然只需要短短的几行代码就搞定了，但是我们对于模型的理解估计不会这么深入。"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
